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Abstract 

Inpainting  is  an  image  interpolation  problem,  with  broad  applications  in  im¬ 
age  and  vision  analysis.  This  paper  presents  our  recent  efforts  in  developing 
inpainting  models  based  on  the  Bayesian  and  variational  principles.  We  discuss 
several  geometric  image  models,  their  role  in  the  construction  of  variational  in¬ 
painting  models,  and  the  associated  Euler-Lagrange  PDEs  and  their  numerical 
computation. 
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1.  Introduction 

The  word  “inpainting”  is  an  artistic  synonym  for  “image  interpolation,” 
as  frequently  used  among  museum  restoration  artists,  who  manually  remove 
cracks  from  degraded  ancient  paintings  by  following  as  faithfully  as  possible 
the  original  intention  of  their  creators  [EM76,  Wal85].  A  mathematical  illus¬ 
tration  is  depicted  in  Figure  1 . 
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As  fine  art  museums  go  digital,  all  real  paintings  are  scanned  into  computers. 
No  doubt,  digital  inpainting  provides  the  safest  way  to  restore  those  degraded 
ancient  paintings,  simply  by  trying  computer  codes  and  softwares  on  the  digi¬ 
tal  copies.  Unlike  the  manual  inpainting  process  which  applies  directly  to  the 
original,  digital  inpainting  brings  tremendous  freedom  in  making  errors  or  im¬ 
proving  results  progressively,  with  no  risk  of  destroying  the  original  precious 
painting  on  the  canvas,  which  is  often  unique  in  the  entire  world. 

But  the  application  of  digital  inpainting  goes  far  beyond  on-line  art  muse¬ 
ums.  In  on-line  real  estate  business,  for  example,  a  potential  customer  may 
ask:  I  do  not  like  the  palm  tree  and  bush  in  the  front  yard;  what  will  the  house 
look  like  without  them?  If  the  tree  and  bush  are  too  close  to  the  house,  it  is 
impossible  to  capture  by  ordinary  cameras  an  unblocked  overview  of  the  en¬ 
tire  house.  However,  treating  the  tree  and  bush  as  an  inpainting  domain,  it  is 
very  hopeful  that  a  (clever)  inpainting  scheme  can  get  rid  of  them  in  a  realistic 
fashion. 


U° \jyc  is  given 


Figure  1.  For  a  typical  inpainting  problem,  the  image  is  missing  on  an  inpainting  domain 
D,  and  the  available  part  u°  |  D<.  is  often  noisy.  D  can  be  disconnected. 

Ever  since  the  original  work  of  Bertalmio  et  al.  [BSCBOO],  digital  inpaint¬ 
ing  has  found  wide  applications  in  image  processing,  vision  analysis,  and  the 
movie  industry.  Recent  examples  include:  automatic  scratch  removal  in  digi¬ 
tal  photos  and  old  films  [BSCBOO,  CSOla],  text  erasing  [BBC+01,  BSCBOO, 
CSOla,  CSOlc],  special  effects  such  as  object  disappearance  and  wire  removal 
for  movie  production  [BSCBOO,  CSOlc],  disocclusion  [MM98],  zooming  and 
super-resolution  [BBC+01,  CSOla,  TAYW01],  lossy  perceptual  image  cod¬ 
ing  [CSOla],  and  removal  of  the  laser  dazzling  effect  [CCBT01],  and  so  on. 
On  the  other  hand,  in  the  engineering  literature,  there  also  have  been  many 
earlier  works  closely  related  to  inpainting,  which  include  image  interpola¬ 
tion  [AKR97,  KMFR95a,  KMFR95b],  image  replacement  [IP97,  WFOO],  and 
error  concealment  [JCF94,  KS93]  in  the  communication  technology. 

As  scattered  as  the  applications  are,  the  methods  for  inpainting  related  prob¬ 
lems  have  also  been  very  diversified,  ranging  from  nonlinear  filtering  method, 
wavelets  and  spectral  method,  and  statistical  method  (especially  for  textures), 
etc. 
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The  most  recent  approach  to  non-texture  inpainting  is  based  on  the  PDE 
method  and  Calculus  of  Variations,  and  can  be  classified  into  two  categories. 
The  first  class  is  based  on  the  simulation  of  micro-inpainting  mechanisms.  It 
includes  the  axiomatic  approach  of  Caselles,  Morel  and  Sbert  [CMS98],  the 
transport  process  [BSCBOO]  (the  first  high-order  PDE  model),  the  diffusion 
process  [CSOlc],  and  their  combination  [BBS01,  CSOlb].  The  second  category 
includes  all  variational  models  simulating  the  unique  macro-inpainting  mecha¬ 
nism:  “best  guess,”  or  the  Bayesian  framework  [GG84,  KR96,  Mum94b].  The 
latter  includes  the  total  variation  model  [CSOld,  CSOla,  R094,  ROF92],  the 
functionalized  elastica  model  [CKS01,  MM98],  the  value-and-direction  joint 
model  [BBC+01],  the  active  contour  model  based  on  Mumford  and  Shah's 
segmentation  [TAYW01],  and  the  inpainting  scheme  based  on  the  Mumford- 
Shah- Euler  image  model  [ES02]. 

The  current  paper  surveys  and  summarizes  this  latter  category.  The  main 
goal  is  to  develop  a  systematic  approach  and  mathematical  foundation  for  all 
these  previously  scattered  works,  so  that  the  survey  can  serve  as  a  fresh  stalling 
point,  rather  than  a  concluding  chapter,  for  further  research  on  this  challenging 
topic. 

The  philosophy  behind  Bayesian  inpainting  is  quite  simple  and  intuitive  (see 
Figure  1):  the  way  we  human  inpainters  inpaint  an  incomplete  picture  mostly 
relies  on  two  factors  —  how  we  read  the  existing  pail  of  the  picture  u°  |  n^D  (i.e. 
data  model),  and  what  class  of  images  we  believe  the  original  good  picture  u 
belongs  to  (i.e.  image  prior  model).  (For  example,  if  it  is  known  that  we  are 
inpainting  an  image  of  a  kitchen  with  tomatoes,  peppers,  and  apples,  we  have 
the  a  priori  preference  of  smooth  shapes  and  the  green  and  red  colors.)  In 
the  Bayesian  language,  a  balanced  optimal  guess  is  to  maximize  the  posterior 
probability  Pmb(u|u°)  (MAP)  given  by 


Prob(u|rt°) 


Prob(u°|u)  •  Prob(ri) 
Prob(u°) 


(1) 


Once  an  image  u°  is  given,  the  denominator  is  a  fixed  constant.  Thus  we  are 
to  maximize  the  product  of  the  data  probability  and  the  image  probability. 

For  inpainting,  the  data  model  is  usually  simple  as  illustrated  in  Figure  1 : 
the  available  pail  u°|o\yj  is  the  restriction  of  the  original  good  image  u  on 
i)\D,  polluted  independently  by  white  noise  n,  i.e. 


In\£> 


=  u 


n\D 


+  71. 


On  the  other  hand,  since  there  is  no  data  available  on  the  inpainting  domain 
D ,  the  task  of  reconstructing  the  image  on  D  solely  falls  on  the  image  model. 
This  makes  a  good  image  model  more  crucial  for  inpainting  than  for  any  other 
classical  restoration  problems  such  as  denoising,  deblurring,  and  segmenta¬ 
tion  [CSOla,  ES02], 
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Image  models  can  be  learned  from  image  data  banks  based  on  filtering, 
parametric  or  non-parametric  estimation,  and  the  entropy  method  (See  Zhu, 
Wu  and  Mumford  [ZM97,  ZWM97]  for  examples).  Such  statistical  approach 
is  especially  important  for  inpainting  or  synthesizing  images  with  rich  tex¬ 
tures  [IP97,  WLOO], 

On  the  other  hand,  in  most  inpainting  problems,  the  inpainting  domain  of¬ 
ten  “erases”  some  perceptually  important  geometric  information  of  the  image, 
edges,  for  example.  To  reconstruct  geometry,  it  is  necessary  that  the  image 
model  well  resolves  the  geometry  a  priori .  Most  conventional  probabilistic 
models  lack  such  feature.  Fortunately,  “energy”  forms  do  exist  in  the  litera¬ 
ture,  which  have  been  explicitly  motivated  by  geometry.  Well-known  examples 
include  the  TV  (total  variation)  model  of  Rudin,  Osher  and  Fatemi  [ROF92] 
and  the  object-edge  model  of  Mumford  and  Shah  [MS89].  The  link  between 
probabilistic  image  models  and  such  geometric  image  models,  as  Mumford 
pointed  out  [Mum94b],  is  formally  made  through  Gibbs’  formula  in  statistical 
mechanics  [Gib02] : 

Prob(tr)  =  exp(— /3E[u]), 

z 

where  E[u]  is  the  energy  of  u  (e.g.,  the  total  variation  of  u),  (3  denotes  the 
inverse  absolute  temperature,  and  Z  the  partition  function.  (Working  with  en¬ 
ergy  also  frees  one  from  laboring  on  the  definability  of  the  partition  function 
Z,  which  is  generally  a  highly  non-trivial  mathematical  issue.)  The  Bayesian 
formula  (1)  is  re-expressed  in  the  energy  form  by 

E[u\u°]  —  E[u°\u\  +  E[u\  +  const., 

where  the  constant  can  be  dropped  safely  as  far  as  energy  minimization  is 
concerned. 

The  organization  goes  as  follows.  Section  2  starts  with  an  axiomatic  ap¬ 
proach  for  curve  models,  which,  to  our  best  knowledge,  is  new.  The  latter 
half  of  the  section  explains  two  approaches  for  constructing  geometric  image 
models  from  curve  models: 

(i)  through  direct  functionalization  based  on  the  level-sets;  and 

(ii)  by  having  a  curve  model  embedded  as  an  edge  model  in  the  object-edge 
primitive  image  model. 

These  approaches  unify  the  four  geometric  image  models  appealing  in  the  re¬ 
cent  inpainting  literature: 

(1)  the  TV  image  model  of  Rudin,  Osher  and  Fatemi  [ROF92,  R094],  first 
applied  to  inpainting  modeling  by  Chan  and  Shen  [CSOla]; 
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(2)  the  functionalized  elastica  image  model  as  proposed  and  studied  by  Mas- 
nou  and  Morel  [MM98],  and  Chan,  Kang,  and  Shen  [CKS01]; 

(3)  the  Mumford-Shah  image  model  [MS89]  applied  to  inpainting  by  Tsai, 
Yezzi,  and  Willsky  [TAYW01],  Chan  and  Shen  [CSOla],  and  Esedoglu 
and  Shen  [ES02];  and 

(4)  the  Mumford-Shah-Euler  image  model  designed  for  image  inpainting  by 
Esedoglu  and  Shen  [ES02], 

Section  3  explains  all  the  recent  inpainting  schemes  based  on  these  geometric 
image  models.  We  discuss  the  associated  Euler- Lagrange  PDE’s,  their  geomet¬ 
ric  meaning,  and  robust  ways  of  numerical  implementation.  Digital  examples 
are  given  for  each  inpainting  model.  Conclusion  and  open  problems  are  written 
into  Section  4. 

Throughout  the  paper,  0  denotes  the  entire  image  domain,  D  the  miss¬ 
ing  inpainting  domain,  u°  the  available  paid  of  the  image  on  it\D,  and  u 
the  targeted  inpainting  restoration.  The  standardized  symbols  V,  V-  and  A 
represent  the  gradient,  divergence,  and  Laplacian  operators  separately.  For 
any  multi- variable  function  or  functional  F(X,Y),  the  symbol  F(X\Y)  still 
means  F(X,  Y),  but  emphasizing  that  Y  is  fixed  as  known.  This  is  to  imitate 
the  symbol  of  conditional  probability  or  expectation  appealing  in  the  Bayesian 
formula  (but  without  probabilistic  normalization). 

2.  Geometric  Image  Models 
2.1  Curve  models 

Geometry  plays  a  crucial  role  in  visual  perception  and  image  understanding, 
including  classification  and  pattern  recognition.  The  most  important  geometry 
for  image  analysis  is  hidden  in  edges.  From  David  Marr’s  classical  work  on  pri¬ 
mal  sketch  [MH80]  to  David  Donoho’s  geometric  wavelets  analysis  [DonOO], 
edges  always  stay  in  the  heart  of  many  issues:  image  coding  and  compression, 
image  restoration,  segmentation  and  tracking,  just  to  name  a  few. 

Therefore  it  is  of  fundamental  significance  to  understand  how  to  mathemat¬ 
ically  model  edges  and  curves. 

From  the  Bayesian  point  of  view,  this  is  to  establish  a  probability  distri¬ 
bution  Prob(T)  over  “all”  curves.  An  instant  example  coming  to  mind  is  the 
Brownian  motion  and  its  Wiener  measure  [KS97].  The  problem  is  that  Brow¬ 
nian  paths  arc  parameterized  curves  (by  “time”)  and  arc  even  almost  surely 
no- where  differentiable.  For  image  analysis,  however,  edges  arc  intrinsic  (1- 
D)  manifolds  and  their  regularity  is  an  important  visual  cue. 

According  to  the  previously  stated  Gibbs’  formulation,  we  are  to  look  for  a 
suitable  energy  form  AfT],  It  is  always  convenient  to  first  start  with  its  digital 
version. 
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In  digital  image  processing.  Freeman's  Chain  Coding  [Fre61]  is  a  popular 
data  structure  for  representing  object  borders  and  edges.  The  underlying  idea 
is  to  represent  a  1-D  curve  T  by  a  chain  of  ordered  sample  points 


>®0 1  1  1  1  5  3CN i 

dense  enough  to  ensure  reasonable  approximation.  Working  directly  with  such 
chains  of  finite  length,  we  need  to  define  appropriate  energy  forms 

E\x o,a?i,---  , ®jv]. 

To  the  best  of  our  knowledge,  the  following  axiomatic  approach  is  new  in  the 
literature.  We  shall  naturally  construct  two  of  the  most  useful  planer  curve 
models:  the  length  energy  and  Euler’s  elastica  energy. 

Axiom  1.  Euclidean  invariance. 

Let  Q  €  0(2)  (conventionally  called  a  rotation,  though  including  all  reflec¬ 
tions),  and  c  e  i?2  an  arbitrary  point.  Euclidean  invariance  consists  of  two 
parts:  the  rotational  invariance 


E\Qx o,  Qx\ ,  ■  ■  ■  ,  Qxn]  —  E\x o,  ®i,  ■  ■  ■  ,  ®jv]j 


and  the  translation  invariance 


E  [«eq  -f -  c ,  a?  2  T  c ,  ■  ■  ■  i  n  Tcj  —  E  x\^ ' ' ' 


Axiom  2.  Reversal  invariance. 
It  requires  that 


E\x o,  ■  ■  ■  —  E\xjs[ ,  •  •  • 

which  means  that  the  energy  does  not  depend  on  the  orientation  of  the  curve. 
Axiom  3.  p-point  accumulation  (p  —  2,  3,  ■  ■  ■ ). 

This  is  fundamentally  a  rule  on  locality.  A  p- point  accumulative  energy 
satisfies  the  accumulation  law: 

E\x o,  ■  ■  ■  ,  xn,  a:n+i]  =  E\x o,  •  •  ■  ,  a?n]  T  E\xn—p+2i ' ' '  i  ®n- i-i]: 
for  all  n  >  p  —  2.  Through  cascade,  we  easily  establish  that 
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Proposition  1  Suppose  E  is  p-point  accumulative.  Then  for  any  N  >  p  —  1, 

N-p+1 

E\x  o,  •  •  •  ,  35  jv]  —  E  Xm  '  '  '  5  ^n+p—  l] 

n= 0 

Thus,  for  example,  a  2-point  accumulative  energy  must  be  in  the  form  of 

E[ as0,  ■  ■  ■  ,  xN]  -  E[ aj0,  ®i]  +  #[®i,  ®2]  H - b  *tv]; 

and  a  3-point  accumulative  energy  satisfies 

E\x o,  ■  ■  •  ,  ®jv]  —  T/[o:o,  *®i,  ®2]  T *r2,  ®3]  T-  ■  •  T E\xn— 2,  xjy— i,  ®jv]- 

Generally,  a  p-point  accumulative  energy  E  is  completely  determined  by  its 
fundamental  form 

E\x 0,  ■  ■  ■  ,  Xp— 1]. 

In  what  follows,  we  study  the  cases  of  p  =  2  and  p  —  3. 

2-point  accumulative  energy  and  the  length  . 

Proposition  2  A  Euclidean  invariant  2-point  accumulative  energy  must  be  in 
the  form  of 

N- 1 

1  ®tv]  —  ^  ^  / (|®n+l  ®n|)) 

n=0 

for  some  non-negative  function  f(s). 

Proof  We  only  need  to  show  that 

E[x 0,®i]  =  /(|®i  -  ®0|). 

Translation  invariance  leads  to 

£[®o,  ®i]  =  E[ 0,  ®i  -  ®0]  =  F(xi  -  ®0), 

with  F(x )  =  ITO.  ®],  Then  rotational  invariance  further  implies  that 

F(Q*)  =  F{x),  Q  E  0(2), *  G  i?2. 

Thus  if  we  define  f(s)  —  F((s,  0)),  then  F(x)  —  /(|®|).  □ 

If  in  addition,  we  impose 

Axiom  4.  Lineal-  additivity: 

For  any  a  E  (0, 1),  and  xi  —  ax 0  +  (1  —  0)0:2, 

E[xq,  xf\  =  E[x0,xi]  +  E[x i,x2\. 
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Then  it  is  easy  to  show  that  the  energy  is  unique  up  to  a  multiplicative  con¬ 
stant. 

Theorem  1  A  Euclidean  invariant  2-point  accumulative  energy  E  with  linear 
additivity  must  be  the  length  energy,  i.  e. , 

n-i 

F [a?o5  ■  ■  ■  ;  ®jv]  —  c  ^  ^  ®n+ 1 1 <f 

n=0 

/or  some  fixed  positive  constant  a 

For  a  summable  curve  T,  as  N  — >  oo,  and  the  sampling  size 

max  \xn  ®n+l| 

0<n<N—l 

tends  to  zero,  such  digital  energy  converges  to  the  length. 

3-point  accumulative  energy  and  the  curvature. 

To  determine  the  fundamental  form  E[xQ,x\,xf\,  first  recall  Frobenius’ 
classical  theorem  [COS+98].  The  three  points  xq,x\,x-2  live  in  R6  —  FI2  x 
R2  x  R2,  and  the  dimension  of  a  Euclidean  orbit  is  3:  1  from  the  rotation  group, 
and  2  from  the  translation  group.  Therefore,  Frobenius’  theorem  applied  to  the 
Euclidean  invariance  gives 

Proposition  3  One  can  find  exactly  three  independent  joint  invariants:  I j .  I2 , 
and  I3,  such  that  E[x  0,  x\,  xf\  is  a  function  of  them. 

Define 


cl  —  |  x\  *^0(5  ^  —  |a?2  *^i|?  c  —  |a?2  xq 


Then  the  ordered  triple  (a,  h,  c )  is  apparently  Euclidean  invariant,  and  two 
chains  [xo,®i,®2]  and  [2/05 2/i 5 2/2]  are  Euclidean  congruent  if  and  only  if 
they  share  the  same  (a,  6,  c).  Thus  there  must  exist  a  non-negative  function 
F(a.  6,  c)  such  that 

E[x  o,aJi,aj2]  =  F(a,b,c). 

Define  the  two  elementary  symmetric  functions  of  a  and  b: 


Ai 


a  +  b 
2 


and  B\  —  ab. 


The  reversal  invariance  implies  the  symmetry  of  F  with  respect  to  a  and  b. 
Thus  E  has  to  be  a  function  of  and  c: 


E[  x0,xi,x2\  =  f{A1,Bllc). 
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Let  s  denote  the  half  perimeter  of  the  triangle  (®i,  x2,  *3): 

s  =  Al  +  r 

and  A  its  area: 

A  =  y/s(s  —  a)(s  —  b)(s  —  c )  =  \/s(s  —  c)(s2  —  2A^s  +  B\). 

Then  we  can  define  the  digital  curvature  at  x\  [COS+98,  COT96,  BouOO]  by 

A  sin#i 


K\  —  4 


B\c  c/2  ’ 


where  0\  is  the  angle  facing  the  side  [aj0,  x‘i\-  It  is  shown  by  Calabi,  Olver,  and 
Tannenbaum  [COT96]  that  for  a  generic  smooth  curve  and  a  fixed  point  xi  on 
it,  as  a,  b  -*  0, 

«i  =  k(®i)  +  0(|6  -  a|)  +  0(a2  +  b2), 
where  K(aji)  is  the  curvature  at  x\. 

Now  it  is  easy  to  see  that  is  a  complete  set  of  joint  invariants  for 

both  the  Euclidean  and  reversal  invariances,  and 

E[x  0,®i,aj2]  =  9{kuM,Bi). 

Therefore,  we  have  proved 


Theorem  2  A  3-point  accumulative  energy  E  with  both  Euclidean  and  rever¬ 
sal  invariances  must  be  in  the  form  of 


N- 1 

B\xq,  •  •  •  ,  ®jv]  —  ^  ^  An,  Bn ). 

n= 1 

Further  notice  that,  as  the  sampling  size  a,  b  —  O(Zi),  h  — >  0  at  a  fixed  point 

*1  g  r, 

Ki  =  0(1),  Ax  =  O(h),  Bx  =  0(h2). 

Applying  Taylor  expansion  to  g  for  the  infinitesimals  A\  and  B\  (assuming 
that  g  is  smooth),  we  have 

g{m,  Ai,Bi)  -  gi{Ki)Ai  +g2{Ki)Bi  H - , 

for  some  functions  51, 52,  ■  ■  •  ■  In  the  linear  integration  theory,  by  neglecting 
all  high  order  (>  2)  infinitesimals,  we  end  up  with 


g{Ki,AuBi)  =  gi{ni)Ai. 
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Therefore  we  have  derived, 

Corollary  1  Following  Theorem  2,  in  addition,  suppose  that  for  any  smooth 
summable  simple  curve  T,  and  its  Chain  Coding  approximation  [rco,  ®i ,  ■  ■  ■  a:  at] 
with  the  size 

h  —  max  rc^-Li  I 

0<n<N—l 

tending  to  zero,  E[x  q,  ■  •  •  Xpf]  converges.  Then  as  far  as  the  limit  is  concerned, 
there  is  only  one  class  of  such  energy,  which  is  given  by 


N-l 

E[xq,  •  •  •  ,  ®jv]  ~  ^  ^  f ■ 

n=  1 


As  h  -»  0,  it  converges  to 

E[T]  =  J  f(n)ds , 
where  ds  is  the  length  element. 

For  instance,  if  we  take  f(n)  —  a  +  fin2  for  two  fixed  weights  a  and  /3, 
the  resulting  energy  is  called  the  elastica  energy  [Mum94a],  which  was  first 
studied  by  Euler  in  modeling  the  shape  of  a  torsion  free  thin  rod  in  1744.  If 
13  —  0,  the  elastica  energy  degenerates  to  the  length  energy. 

2.2  Image  models  via  functionalizing  curve 
models 

Once  a  curve  model  i7[r]  is  established,  it  can  be  “lifted”  to  an  image  model 
by  direct  functionalization  and  the  level-set  approach. 

Let  u{x)  be  an  image  defined  on  a  domain  il  C  R2.  For  the  moment  assume 
that  u  is  smooth  so  that  almost  surely  for  each  gray  value  A,  the  level-set 

=  {x  €  :  u(x)  —  A} 

is  a  smooth  1-D  manifold.  Let  to  (A)  be  an  appropriate  non-negative  weight 
function.  Then  based  on  a  given  curve  model  _E[T],  we  can  construct  an  image 
model: 

/OO 

£[TA]to(A)dA. 

-OO 

Conventionally  to  (A)  is  set  to  1  to  reflect  human  perceptual  sensitivity.  Sup¬ 
pose  we  have  a  bundle  of  level-sets  whose  gray  values  arc  concentrated  over 
[A,  A  +  AA].  If  A  A  is  small,  then  the  image  appeal's  smooth  over  the  region 
made  of  these  level-sets,  and  is  thus  less  sensitive  to  perception.  The  energy 
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assigned  to  such  bundles  should  be  small  accordingly.  On  the  other  hand,  if 
AA  is  large,  for  example  in  the  situation  when  the  bundle  contains  a  sharp 
edge,  then  the  level-sets  carry  important  visual  information  and  the  associated 
energy  should  be  large.  Therefore,  the  Lebesgue  measure  dX  is  already  per¬ 
ceptually  motivated  and  w(X)  can  be  set  to  1,  which  we  shall  assume  in  the 
following. 

Suppose  we  take  the  length  energy  in  Theorem  1  as  the  curve  model,  then 
the  resulting  image  model 


length  (T^)dA 


is  exactly  Rudin-Osher-Fatemi’s  TV  model  [ROF92,  R094]: 

E[u]  —  /  \Vu\dx. 

Jn 

This  is  because  for  a  smooth  image  u,  along  any  level-set  T \, 
dX  —  \Vu\da,  length^)  =  /  ds , 

J  rA 

with  ds  and  da  denoting  the  arc  lengthes  of  the  level-sets  and  gradient  flows, 
which  are  orthogonal  to  each  other,  and  thus 


dsda  —  dx 


is  the  area  element.  Therefore, 

/OO 

-OO 

The  above  derivation  is  in  a  formal  level  and  can  be  rigorously  established 
based  on  the  theory  of  BV  functions  [Giu84],  where  the  length  of  a  level-set  is 
replaced  by  the  perimeter  of  its  associated  region,  the  Sobolev  gradient  norm 
by  the  TV  radon  measure.  Then  the  lifting  process  is  exactly  the  famous  co¬ 
area  formula. 

Similarly,  suppose  we  take  the  curvature  curve  model  in  Corollary  1,  then 
the  lifted  image  model  becomes 

lVUldX- 

Especially,  if  f(s)  —  a  +  (3s 2,  it  is  called  the  elastica  image  model  [CKS01, 
MM98]. 
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2.3  Image  models  with  embedded  edge  models 

The  second  approach  to  construct  image  models  from  curve  models  is  based 
on  the  object-edge  primary  model,  as  proposed  by  Mumford-Shah  [MS89].  In 
such  image  models,  the  curve  model  is  embedded  to  weigh  the  energy  from 
the  edges,  i.e.,  abrupt  jumps  in  images. 

For  example,  the  classical  Mumford-Shah  image  model  employs  the  length 
curve  model: 

E[u,  T]  =  /  \Vu\2dx  +  cdength(r). 

Jn\r 

Here  T  denotes  edge  collection.  Unlike  the  TV  image  model,  once  the  singular 
set  T  is  singled  out,  for  the  rest  of  the  image  domain,  Sobolev  smoothness  can 
be  legally  imposed. 

Mumford-Shah  image  model  has  been  very  successful  in  image  segmenta¬ 
tion  and  denoising.  For  image  inpainting,  as  Esedoglu-Shen  discussed  in  [ES02], 
it  is  intrinsically  insufficient.  Therefore  a  new  image  model  called  Mumford- 
Shah-Euler  is  proposed  in  [ES02]  based  on  the  elastica  curve  model: 

E[u,  T]  =  /  \Vu\2dx  +  j{a  +  f3K2)ds. 

Jn\r  Jt 

We  now  start  to  discuss  how  to  carry  out  inpainting  based  on  these  image 
models. 

3.  Inpainting  Models  and  Their  PDE’s 

In  this  section,  we  survey  all  the  recent  inpainting  schemes  based  on  the  ge¬ 
ometric  image  models  mentioned  above.  We  shall  describe  the  PDE  forms  for 
all  the  variational  models,  and  their  digital  realization  based  on  the  numerical 
PDE  method.  Digital  examples  are  demonstrated  for  each  inpainting  scheme. 

3.1  The  TV  inpainting 

In  [CSOla],  we  first  touched  on  the  Bayesian  idea  for  the  inpainting  prob¬ 
lem,  as  an  alternative  to  the  PDE  approach  invented  by  Bertalmio  et  al.  [BSCBOO] 
based  on  the  transport  mechanism.  The  image  model  employed  in  [CSOla]  is 
the  well-known  Rudin-Osher-Fatemi's  TV  image  model,  as  first  proposed  for 
the  denoising  and  deblurring  application  [ROF92,  R094]. 

The  TV  inpainting  model  is  to  minimize  the  posterior  energy 

J(;V[rt|w0, D]  —  (  \Vu\dx  +  —  f  (u  —  u°)2dx.  (2) 

Jn  2  Jq\d 

Define 

A d(x)  =  A  •  ln\D{x). 
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Then  the  steepest  descent  equation  for  the  energy  is 

du  T  Vu  1  ,  .  w  n  . 

at  =  v’|jvqJ +  AbM(u  -“>■  <3) 

which  is  a  diffusion-reaction  type  of  nonlinear  equation.  To  justify  the  drop 
of  the  boundary  integral  coming  from  the  variational  process,  the  associated 
boundary  condition  along  Oil  is  adiabatic:  du/dV  —  0,  where  V  denotes  the 
normal  direction  of  the  boundary. 

The  diffusion  is  anisotropic  to  respect  sharp  edges  as  in  the  Perona-Malik 
diffusion  [PM90]  since  the  diffusivity  coefficient  1  / 1  Vu  becomes  small  where 
u  has  sharp  jumps.  The  reaction  term  has  u°  as  its  attractor  to  keep  the  solution 
close  to  the  given  noisy  image.  But  notice  that  on  the  inpainting  domain  D , 
the  equation  is  a  pure  diffusion  process,  which  of  course  originally  comes  from 
the  TV  image  model. 

In  [CKS01],  the  existence  of  a  minimizer  of  Jtv  in  the  BV  space  is  estab¬ 
lished  based  on  the  direct  method  of  Calculus  of  Variation.  The  uniqueness, 
however,  is  generally  not  guaranteed.  An  example  is  given  in  [CKS01].  Non¬ 
uniqueness,  from  the  vision  point  of  view,  reflects  the  uncertainty  of  human 
visual  perception  in  certain  situations,  and  thus  should  not  be  cursed  in  terms 
of  faithful  modeling. 

For  the  digital  realization  of  model  (3),  the  degenerate  diffusion  coefficient 
1/|  Vu  |  is  always  conditioned  to 


for  some  small  positive  constant  a.  From  the  energy  point  of  view,  it  amounts 
to  the  minimization  of  the  modified  Jtv : 

JtvM  =  [  !VuU  +  T  /  ~~  u°)2dx.  (4) 

Jn  z  Jn\D 

This  energy  form  connects  image  inpainting  to  the  classical  problem  of  non- 
parametric  minimal  surfaces  [Giu84].  In  fact,  in  the  ( x,y,z )  space,  the  first 
term  of  J'f,  [u] ,  up  to  the  multiplicative  constant  a,  is  exactly  the  area  of  the 
surface 

z  =  z(x,y)  =  u(x,  y)j a. 

In  the  case  when  the  available  paid  u°  is  noise-free,  we  have 

X  —  oo,  =  ^°|n\£). 

Thus  we  end  up  with  the  exact  minimal  surface  problem  on  the  inpainting 
domain  D : 

min  /  \J  1  +  \S7z\2dx  with  z  —  z°  along  dD. 

Jd 
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Here  along  the  boundary,  z \QD  is  understood  as  the  trace  from  the  interior. 
Since  this  Dirichlet  problem  might  not  be  solvable  for  general  inpainting  do¬ 
mains  D  (see  [Giu84]  for  example),  as  far  as  inpainting  is  concerned,  we  may 
formulate  a  weaker  version  even  for  the  noise-free  case: 

min  [  yfl  +  |V*|2  +  £  [  (z  —  z°)2dH1, 

J  D  *  JdD 


where  /i  is  a  large  positive  weight  and  dH\  the  1 -dimensional  Hausdorff  mea¬ 
sure  of  d D.  Then  the  existence  of  a  minimum  can  be  easily  established  based 
on  the  direct  method. 

Compared  with  all  other  variational  inpainting  schemes,  the  TV  model  has 
the  lowest  complexity  and  easiest  digital  implementation.  It  works  remarkably 
well  for  more  local  inpainting  problems  such  as  digital  zoom-in  (Figure  2) 
and  text  removal  [CSOla],  But  for  large-scale  inpainting  problems,  the  TV 
inpainting  model  suffers  from  its  origin  in  the  length  curve  energy.  One  major 
drawback  is  its  failure  to  realize  the  Connectivity  Principle  in  visual  perception 
as  discussed  in  [CSOlc], 


The  original  image  Zoom-out  by  a  subsampling  of  factor  4 


The  harmonic  zoom-in 


W  : 

% 

• 

n 

The  TV  zoom-in 


Figure  2.  Digital  zoom-in  based  on  the  TV  inpainting  scheme,  as  compared  with  that  based 
on  the  harmonic  inpainting  scheme,  i.e.,  that  based  on  the  Sobolev  image  model:  E[u]  — 

/  | S7u\2dx.  Notice  that  the  TV  gives  much  shaiper  boundary  reconstruction. 

J  n 
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The  original  image 


Edge  tube  from  Canny’s  detector 


The  initial  guess 


The  TV  inpainting 


Figure  3.  TV  inpainting  applied  to  the  primal-sketch  based  image  decoding. 


3.2  The  elastica  inpainting 

In  [CKS01],  Chan,  Kang,  and  Shen  proposed  to  improve  the  TV  inpainting 
model  by  using  the  elastica  image  model 

E[u]  =  /  (a  +  f3n2)dx ,  k  —  V  • 

Jn 

The  elastica  inpainting  model  is  thus  to  minimize  the  posterior  energy 

Je[u\u°,D]—  f  (j)(n)dx  +  ^  f  (rt  —  tt°)2efe,  (5) 

J o  *  Jn\D 

where  (f)(s )  —  a  +  f3s2. 

By  Calculus  of  Variation,  it  is  shown  in  [CKS01]  that  the  steepest  descent 
equation  is  given  by 


Vu  ' 

W\. ' 


i  —* 

—  =  V  •  V  +  A D(x)(u°  -  u), 


V  —  (f)(n)n 


t  d(<fi'(K)\Vu\) 


|V«| 


dt 


Here  n.  t  arc  the  normal  and  tangent  directions: 


Vu 


t  —  n±, 


(6) 

(V) 


n  — 


16 


Notice  that  the  coupling  of  t  and  d/dt  in  (7)  makes  it  safe  to  take  any  direction 
of  nL  for  t.  The  natural  boundary  conditions  along  d£l  arc 


and 


d{ff{K)\Vu\) 

du 


The  vector  field  V  is  called  the  flux  of  the  elastica  energy.  Its  decomposition 
in  the  natural  orthogonal  frame  (n.  t )  in  (7)  has  significant  meaning  in  terms 
of  micro-inpainting  mechanisms. 


(i)  The  normal  flow  ('p(k)n  carries  the  feature  of  an  important  inpainting 
scheme  invented  earlier  by  Chan  and  Shen  called  CDD  ( curvature  driven 
diffusion )  [CSOlc].  CDD  was  discovered  in  looking  for  micro  mecha¬ 
nisms  that  can  realize  the  Connectivity  Principle  in  visual  perception  [CSOlc, 
Kan79,  NMS93]. 


(ii)  The  tangential  component  can  be  written  as 


(  1 

\  |  Vrt|2  dt  ) 


and  its  divergence  is 


V  •  Vt  =  VLU  •  V 


(  -1 

V|Vrr|2  dt  ) 


since  VLu  is  divergence  free.  Define  the  smoothness  measure 


-1  d{ff{K)\X7u\) 

*  \Vu\2  dt 

Then  the  tangent  component  is  in  the  form  of  the  transport  inpainting 
mechanism  as  originally  invented  by  Bertalmio  et  al.  [BSCBOO]. 


Pure  transport  can  lead  to  shocks  as  in  traffic  models,  while  pure  curvature 
driven  diffusion  (CDD)  is  only  motivated  by  the  Connectivity  Principle  in  vi¬ 
sion  research  and  lacks  theoretical  support.  The  elastica  inpainting  PDE  (6) 
combines  their  strength  and  also  offers  a  theoretical  framework. 

For  the  numerical  realization  of  the  model,  we  mention  the  following  as¬ 
pects.  More  detail  can  be  found  in  [CKS01].  Two  digital  examples  are  illus¬ 
trated  in  Figure  4. 


(a)  To  accelerate  the  convergence  of  the  steepest  descent  marching  (6)  toward 
its  equilibrium  solution,  one  can  adopt  the  Marq uina-Oshcr  method  [M099] 
by  adding  a  non-negative  “time  correcting  factor”  T{u ,  |  Vu  | ) : 

-  (V-y  +  AD0r)(rro-W)). 
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For  instance,  take  T  —  |  Vu|.  As  shown  in  [M099],  such  simple  technique 
can  substantially  improve  the  numerical  marching  size  and  speed  up  the 
convergence. 

(b)  As  in  the  TV  inpainting  model,  for  the  computation  of  n  and  V,  1  / 1  Vu  is 
always  conditioned  to  l/|Vw|a  to  avoid  a  zero  denominator. 

(c)  To  more  efficiently  denoise  and  propagate  sharp  edges,  classical  numeri¬ 
cal  techniques  from  computational  fluid  dynamics  (CFD)  can  be  very  use¬ 
ful,  including  those  originally  designed  for  capturing  shocks.  Techniques 
adopted  in  [CKS01]  are  the  upwind  scheme  and  the  min-mod  scheme  [OR90] 


TV  Inpainting 


Original  Image  Inpainting  Domain 


Figure  4-  Two  examples  of  elastica  inpainting,  as  compared  with  TV  inpainting.  In  the 
case  of  large  aspect  ratios  [CSOlc],  the  TV  inpainting  model  fails  to  comply  to  the  Connectivity 
Principle. 


з. 3  Inpainting  via  Mumford-Shah  image  model 

The  idea  of  applying  the  Mumford-Shah  image  model  to  inpainting  and 
image  interpolation  first  appeared  in  Tsai,  Yezzi,  and  Willsky  [TAYW01],  and 
Chan  and  Shen  [CSOla],  and  has  been  recently  studied  again  by  Esedoglu  and 
Shen  [ES02]  based  on  the  T-convergence  theory. 

The  model  is  to  minimize  the  posterior  energy 

Jms[fq r|w°, D]  —  ^  [  | Vrt|2cfe+a;length(r)-|-^  f  (u—u°)2dx,  (8) 

2  y0\r  2  Jq\d 

where  7,  a,  and  A  are  positive  weights.  Notice  that  if  D  is  empty,  i.e.  there  is 
no  spatially  missing  domain,  then  the  model  is  exactly  the  classical  Mumford- 
Shah  denoising  and  segmentation  model  [MS  89].  Also,  notice  that  unlike  the 
previous  two  models,  it  outputs  two  objects:  the  completed  and  cleaned  image 

и,  and  its  associated  edge  collection  T. 
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For  a  given  edge  layout  T,  variation  of  Jms  [u|T,  u°,  D]  gives 

7  Aw  +  Ad(x)(u°  —  u)  —  0  on  fi\T,  (9) 

with  the  natural  adiabatic  condition  du/dv  —  0  along  both  T  and  <9Q. 

Denote  the  solution  to  the  elliptic  equation  (9)  by  wr-  Then  the  steepest 
descent  infinitesimal  move  of  T  for  Jms[T|wr,  w°,  D]  is  given  by 

^  +  ^|Vwr|2  +  -^(^r  -u0)2  'j  n.  (10) 

Here  x  €  T  is  an  edge  pixel  and  ft  the  normal  direction  at  x.  The  symbol  [g]  \ 
denotes  the  jump  of  a  scalar  field  g(x)  across  T: 

[p]r(a:)  =  lim  (g(x  +  an)  -  g(x  -  an)). 

o-^0+ 

The  sign  of  the  curvature  n  and  the  direction  of  the  normal  n  arc  coupled  so 
that  nn  points  to  curvature  center  of  T  at  x. 

Note  that  the  curve  evolution  equation  (10)  is  a  combination  of  the  mean 
curvature  motion  [ES91] 

dx/dt  —  ann 

and  a  field-driven  motion  specified  by  the  second  term.  The  field-driven  mo¬ 
tion  attracts  the  curve  toward  the  expected  edge  set,  while  the  mean  curvature 
motion  makes  sure  that  the  curve  does  not  develop  ripples  and  stays  smooth. 

Like  the  TV  inpainting  model,  inpainting  based  on  the  Mumford-Shah  im¬ 
age  model  is  of  second  order.  But  the  extra  complexity  comes  from  its  free 
boundary  nature.  In  [CSOla,  TAYW01],  the  level-set  method  of  Osher  and 
Sethian  [OS88]  is  proposed. 

In  the  most  recent  work  by  Esedoglu  and  Shen  [ES02],  a  simpler  numeri¬ 
cal  scheme  is  developed  based  on  the  T-convergence  theory  of  Ambrosio  and 
Tortorelli  [AT90,  AT92], 

In  the  T-convergence  theory,  the  1 -dimensional  edge  T  is  approximately 
represented  by  its  associated  signature  function 

z  \  f  [0, 1], 

which  is  nearly  1  almost  everywhere  except  on  a  narrow  (specified  by  a  small 
parameter  e)  tubular  neighborhood  of  T,  where  it  is  close  to  0.  The  posterior 
energy  Jms[rq  T|w,°,  D ]  is  approximated  by: 

Jt[u,z\u°,D] f  X d(x)(u  —  u°)2dx  +  ^-  [  z2\Vu\2dx 
2  in  2  in 

+  “X(e|Vz|2  +  li 4T-) 


dx. 


(11) 
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Taking  variation  on  u  and  z  separately  yields  the  Euler-Lagrange  system: 


A d{x)(u  —  u°)  —  7V  •  ( z2Vu )  =  0  (12) 

(j\Vuf)z  +  a  ^-2eA .z  +  ~  (13) 

with  the  natural  adiabatic  boundary  conditions  along  rif 1  (due  to  the  boundary 
integrals  coming  from  integration -by- parts): 


Define  two  elliptic  operators  acting  on  u  and  z  separately: 


Lz  —  —V  •  z2V  +  Xd/'I 
Mu  =  (1  +  2(e7/o:)|Vti|2)  -  4e2A. 


(14) 

(15) 


Then  the  Euler-Lagrange  system  (12)  and  (13)  is  simply  written  as 

Lzu  —  (A  d/i)u°  and  Muz  —  1.  (16) 


This  coupled  system  can  be  solved  easily  by  any  efficient  elliptic  solver  and  an 
iterative  scheme.  Two  digital  examples  are  included  in  Figure  5  and  6. 


Noisy  image  to  be  inpainted 


Inpainting  output  u  Inpainting  output  z 


Figure  5.  Inpainting  based  on  the  T-convergence  approximation  (11)  and  its  associated 
elliptic  system  (16). 


3.4  Inpainting  via  Mumford-Shah-Euler  image 
model 

Like  the  TV  image  model,  the  Mumford-Shah  image  model  is  insufficient 
for  large-scale  image  inpainting  problems  due  to  the  embedded  length  curve 
energy.  To  improve,  Esedoglu  and  Shen  [ES02]  recently  proposed  the  inpaint¬ 
ing  scheme  based  on  the  Mumford-Shah-Euler  image  model. 
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Image  to  be  inpainted 


Inpainting  domain  (or  mask) 


je  Penguin 
l  guys 

c  that  so  many 
ide  a 
{ image 

Itionlos 

» trae?  We 

Ifagree.  We  dre 
ore  optimisti^The  g 


Hello!  We  are  Penguin 
AandB.  You  guys 
must  think  that  so  many 
words  have  made  a 
large  amount  of  image 
information  lost. 

Is  this  true?  We 
disagree.  We  are 
more  optimistic.  The 


Inpainting  output 


Figure  6.  Text  erasing  by  inpainting  based  on  the  Mumford-Shah  image  model. 


In  this  model,  the  posterior  energy  to  be  minimized  is 

^mse[^,r|tt°,D]  =  ^  [  \Vu{2dx  +  [  ( a  +  /3K2)ds  +  ^  f  (u-u°)2dx, 
2  Jn\r  Jr  2  Jn\D 

(17) 

where  the  length  energy  in  Jm s  has  been  upgraded  to  Euler’s  elastica  energy. 

As  in  the  previous  inpainting  model,  for  a  given  edge  layout  T.  the  Euler- 
Lagrange  equation  for  JniseMT,  u°,  D]  is 

'/Au  +  \d(x)(u°  —  u)  —  0,  x  e  fl\r,  (18) 


with  the  adiabatic  condition  along  Y  and  Of l:  du/dv  —  0. 

For  the  solution  uy  to  this  equation,  the  infinitesimal  steepest  descent  move 
of  T  is  given  by  [CKS01,  Mum94a,  LS84]: 


dx 

dt 


a/d2K  n 
cm  -  /3(2^2  +  + 


^|Vur|2  +  ~y(u r  -  u°)2 


J  r 


(19) 


The  meaning  of  the  symbols  is  the  same  as  in  the  previous  section. 

The  digital  implementation  of  this  4th  order  nonlinear  evolutionary  equa¬ 
tion  is  highly  non-trivial.  The  challenge  lies  in  finding  an  effective  numerical 
representation  of  the  1 -dimensional  object  T,  and  robust  ways  to  compute  its 
geometry,  i.e.,  the  curvature  and  its  differentials. 

In  Esedoglu  and  Shen  [ES02],  the  equation  is  numerically  implemented 
based  on  the  T-convergence  theory  of  De  Giorgi  [Gio61],  As  for  the  previous 
Mumford-Shah  image  model,  T-convergence  approximation  leads  to  simple 
elliptic  systems  that  can  be  solved  efficiently  in  computation. 

De  Giorgi  [Gio61]  proposed  to  approximate  Euler’s  elastica  curve  model 

e(T)  =  J  (a  + /3K2)ds , 
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by  an  elliptic  integral  of  the  signature  z  (the  two  constants  a  and  (3  may  vary): 


Et 


46  ) 


2 

dx.  (20) 


where  W (z)  can  be  the  symmetric  double-well  function 

W(z)  =  (1  -z2f  =  (z  +  lf(z-  l)2.  (21) 

Unlike  the  choice  of  W(z)  —  (1  —  z)2  for  the  Mumford-Shah  image  model, 
here  the  edge  layout  V  is  embedded  as  the  zero  level-set  of  z.  Asymptotically, 
as  e  -»  0+,  a  boundary  layer  grows  to  realize  the  sharp  transition  between  the 
two  well  states  z  —  1  and  z  —  —  1. 

Then  the  original  posterior  energy  Jmse  on  u  and  V  can  be  replaced  by  an 
elliptic  energy  on  u  and  z: 


Je[u,z\u°,D\  =  1  [  z2 |V 
1  Jn 


ufdx  +  EJz]  + 


UAd(’ 


u  —  u°)2dx.  (22) 


For  a  given  edge  signature  z,  valuation  on  u  in  Jt[u\z.  u°,  D]  gives 


XD{u-u°)  -7V-  (z2Vu)  =  0, 


(23) 


with  the  adiabatic  boundary  condition  du/dV  ~  0  along  Oil.  For  the  solution 
tt,  the  steepest  decent  marching  of  z  for  Je[z\u,  u°,  D]  is  given  by 


dz 

dt 


7|Vrr|2£  +  ag  + 


PW"{z) 
2  e2 


g  -  4:pAg, 


g  —  2eA^  — 


W'{z) 

4e 


(24) 

(25) 


again  with  the  Neumann  adiabatic  conditions  along  the  boundary  Oil: 


and 


Eq.  (24)  is  of  fourth-order  for  z,  with  the  leading  head  —8ef3A2z.  Thus,  to 
ensure  stability,  an  explicit  marching  scheme  wouldrequire  At  —  0((Ax)4 /e/3) 
There  arc  a  couple  of  ways  to  stably  increase  the  marching  size.  First,  as  in¬ 
spired  by  Marquina  and  Osher  [M099],  one  can  add  a  time  correcting  factor 
(as  in  Section  3.2): 


dz 

dt 


T{Vz,g\u) 


7|Vti|2^  +  ag  + 


I 3W”(z ) 
2e2 


g  -  A/3Ag 


7 


where  T( Vz,  3,  |it)  is  a  suitable  positive  scalai-,  for  example,  T  =  |  Vz\  [M099]. 
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The  second  alternative  is  to  turn  to  implicit  or  semi-implicit  schemes.  Eq.  (24) 
can  be  rearranged  to 

^  +  ^\Vu\2z  -  2aeAz  +  8peA2z  =  ~^W'(z)  +  +  jAW'(z), 

(26) 

Or  simply 


dz 

m 


+  Luz  =  f{z), 


where  Lu  denotes  the  positive  definite  elliptic  operator  ('u- dependent) 
Lu  —  7|Vu|2  —  2aeA  +  8/3eA2, 


and  f(z)  the  entire  right  hand  side  of  (26).  Then  a  semi-implicit  scheme  can 
be  designed  as:  at  each  discrete  time  step  n, 

(1  +  A  tLu)z(n+V  =  *(")  +  A  f/(^(n)), 

where  the  positive  definite  operator  1  +  AtLu  is  numerically  inverted  based  on 
many  fast  solvers  [G092,  Str93] .  A  digital  example  is  given  in  Figure  7. 


A  noisy  image  to  be  inpainted.  Inpainting  via  Mumford-Shah-Euler  image  model 


Figure  7.  Inpainting  based  on  the  Mumford-Shah-Euler  image  model  can  satisfactorily 
restore  a  smooth  edge  as  expected. 

4.  Conclusion  and  open  problems 

In  this  paper,  we  have  surveyed  all  the  recent  inpainting  models  based  on  the 
combination  of  the  Bayesian  principle  and  geometric  image  models.  As  for  the 
classical  denoising,  deblurring,  and  segmentation  applications,  the  Bayesian 
framework  has  proven  to  be  very  effective  in  designing  and  improving  general 
inpainting  models. 
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We  have  explained  that  the  fundamental  ingredient  for  a  geometric  image 
model  is  the  associated  or  embedded  curve  model.  Based  on  some  natural 
axioms  such  as  the  Euclidean  invariance  and  reversal  invariance,  we  have  been 
able  to  understand  the  general  structure  of  geometric  curve  model  on  the  2- 
dimensional  image  domain.  We  have  described  two  general  ways  for  “lifting” 
a  curve  model  to  a  geometric  image  model. 

We  have  observed  that  conventional  first-order  geometric  image  models, 
such  as  the  TV  model  and  Mumford-Shah  model,  function  very  well  for  classi¬ 
cal  denoising,  deblurring,  and  segmentation  problems,  as  well  as  for  inpainting 
problems  with  a  more  local  nature  (such  as  zoom-in  and  text  erasing).  But  for 
large-scale  inpainting  problems,  they  arc  insufficient  for  reconstructing  per¬ 
ceptually  meaningful  outputs.  Therefore,  high  order  geometric  image  models, 
such  as  the  elastica  model  and  the  Mumford-Shah-Euler  model,  become  nec¬ 
essary  for  more  general  inpainting  applications.  The  tradeoff  is  that  high  order 
inpainting  models  are  computationally  much  more  challenging. 

We  have  described  all  the  Euler-Lagrange  PDE’s  associated  to  these  models, 
their  geometric  meaning,  and  their  digital  realization  based  on  techniques  from 
numerical  PDE’s  and  the  T-convergence  theory. 

Finally  we  post  three  interesting  open  problems. 

(i)  Video  inpainting.  Video  inpainting  has  profound  application  in  the  movie 
industry,  surveillance  analysis,  and  dynamic  vision  analysis.  The  first 
open  problem  is:  how  to  integrate  the  extra  dimension  of  “time”  into  the 
spatial  inpainting  techniques?  And  how  to  define  geometric  prior  models 
for  spatial-temporal  images? 

(ii)  Texture  inpainting.  Textures  by  definition  arc  image  patterns  with  rich 
statistical  features.  Geometric  image  models  can  well  describe  the  bound¬ 
aries  of  different  texture  patches,  but  arc  apparently  insufficient  for  in¬ 
painting  the  textures  themselves.  Therefore,  the  second  open  problem 
is:  how  to  integrate  geometric  image  models  and  statistical  texture  mod¬ 
els?  And  how  to  grow  textures  through  texture  synthesis  without  creating 
artificial  boundaries? 

(iii)  Fast  and  efficient  digital  realization.  Throughout  this  survey,  numerical 
PDE  has  been  a  core  computational  tool  for  all  the  geometric  inpainting 
models.  The  third  open  problem  concerns  fast  and  efficient  digital  im¬ 
plementation  of  the  associated  PDE’s,  especially  for  the  high  order  ones. 
There  are  a  number  of  non-trivial  questions  that  wait  to  be  answered: 
How  to  develop  discretization  schemes  that  respect  geometry,  the  curva¬ 
ture  and  its  differentials,  for  examples?  How  to  speed  up  convergence 
based  on  various  numerical  techniques  such  as  the  multigrid  method  and 
the  multiresolution  decomposition?  since  speed  is  always  highly  con¬ 
cerned  in  applications.  Finally,  the  energies  of  high  order  inpainting 
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models,  such  as  the  elastica  and  the  Mumford-Shah-Euler  models,  of¬ 
ten  have  many  local  energy  wells.  It  is  thus  another  important  issue  to 
develop  numerical  schemes  that  can  efficiently  avoid  being  trapped  in 
local  energy  wells,  as  in  Molecular  Dynamics  [MW97], 
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